Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | Go Back
View Post [edit]

Poster: aronsson Date: Oct 3, 2009 4:47am
Forum: forums Subject: Stale .pureftpd-upload* file blocks checkin

During checkin:

Checking item dir...
One or more files (.pureftpd-upload.4ac5db08.15.73c7.99ef6ec4 -- size: 48.9MB) appears to still be FTP transferring -- please let it finish first and then reload this page

This FTP transfer was interrupted and restarted. But I can't remove that .pureftpd-upload* file because it's name is protected. Now my checkin is blocked by this message.

This happened to both
http://www.archive.org/checkin/wikipedia_visitor_stats_200902
and
http://www.archive.org/checkin/wikipedia_visitor_stats_200903

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Oct 5, 2009 3:00pm
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

both fixed. you may want to ensure they were not files partially uploaded that you made sure got reuploaded first?

(sometimes our pure-ftpd server/disks slow a lot -- but i'm not sure why they didn't eventually "finish out" these files...)

--tracey, archive staff

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffHunterNetlabel Date: Feb 11, 2011 11:17am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

Please, I need to check in this item:
http://www.archive.org/checkin/hnr004

It already happened to me before, but I deleted the other item...
But I realy need this one, thank you very much...

Reply to this post
Reply [edit]

Poster: aronsson Date: Oct 6, 2009 7:01am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

Thanks! These two items have now been checked in as
http://www.archive.org/details/wikipedia_visitor_stats_200902
and
http://www.archive.org/details/wikipedia_visitor_stats_200903

I'm still puzzled by the low throughput on FTP uploads. I never see more than 319 kbyte/s per session, but I can double that by running two parallel sessions. Is somebody looking at that?

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffNemo_bis Date: Nov 15, 2010 12:22pm
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

> I'm still puzzled by the low throughput on FTP uploads. I
> never see more than 319 kbyte/s per session, but I can
> double that by running two parallel sessions. Is somebody
> looking at that?

I tried and used Flash but it's even worse: http://www.archive.org/post/333852/maximum-upload-speed-flash
Now I'm uploading with FTP: 9 parallel uploads (which seem to be the maximum, and yet enough to be often flooded with ECONNRESET error) are now at about 1,5 MiB/s on average, with peaks of 2 MiB/s and much more rarely 3 MiB/s, which may be the actual limit on my end (although it should be higher).

This post was modified by Nemo_bis on 2010-11-15 20:22:58

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffNemo_bis Date: Nov 17, 2010 6:43am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

Well, now FileZilla has gone crazy for some minutes because after a "time expired for connection" error which aborted the uploads it couldn't recconnect because of ECONNRESET errors and a "421 Too many connections (10) from this IP" error; I've raised the time between failed login retries from 10 to 60 s and it has recovered after three minutes. Now trying with 8 parallel uploads.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Oct 6, 2009 10:02am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

there can be massive contention for the disks with heavy fragmented reading especially at times. so the speed can be be very bad at times.

we have mostly given up on FTP technology here at archive for exactly the reasons you point out -- why is it going so slow when i can add a second uploader and get 2x the throughput? (that *is* interesting 8-) why are the disks not maxxed out fully by the ftp application? why does resumption not work? all these questions and more were unanswerable by myself and our staff so we have made our single remaining FTP "WIGIWYG", unfortunately ("what you get is what you get").

nonideal to be sure -- but our pure-ftpd server program has had so many problems that we've stopped trying to fix and/or patch around it.

another option is S3 (HTTP PUT) that we support...
-tracey

Reply to this post
Reply [edit]

Poster: kzns.pl Date: Oct 26, 2009 8:56am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

Hi.
Is there a way to automatic upload of a file to add to an item I've already created? (ie use external program to upload)

We have a community radio (internet based) and are give a .wav file recorded by journalists, and local artists.

We have an in-house specialist computer program which with one-click takes the file, converts it to .mp3, adds ID3 tags, adds it to our database, converts it to text, and creates a webpage for it and (up to now) uploads the mp3 to a server (so the non-technical staff do not have to waste their time doing what a computer can do). Also it means technical staff can concentrate on more taxing issues than upload a file.

We would like to use the Internet Archive for this as it is the best or proper place for it for archiving purposes, but... what technical methods exist, however complex, for automating the process of getting that mp3 file from an inhouse computer to a published and accessible version on Internet Archive?

There were some methods floating around (using php) to upload uears ago when the FTP was all working well.


kzns.pl

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Nov 2, 2009 6:30pm
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

yes there is -- i can try to put you in touch w/ an engineer here who can point you to S3. you'd want something like curl (ideally) or wget on a unix-like cmd-line if that's possible. then it's relatively straightforward from there.

we can proceed if that sounds good/plausible and you reply!
-tracey

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Nov 2, 2009 6:33pm
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

yes there is -- i can try to put you in touch w/ an engineer here who can point you to S3. you'd want something like curl (ideally) or wget on a unix-like cmd-line if that's possible. then it's relatively straightforward from there.

we can proceed if that sounds good/plausible and you reply!
-tracey

Reply to this post
Reply [edit]

Poster: aronsson Date: Oct 6, 2009 10:18am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

My friends mentioned the TCP window scaling option. With my ping roundtrip time of 194 ms (from 130.236.254.105 in Sweden) TCP without window scaling might max out at 320 kbyte/s.

It is on at my end. I'm running a standard Ubuntu Linux 9.04 and "cat /proc/sys/net/ipv4/tcp_window_scaling" answers "1" (meaning true/on).

Reply to this post
Reply [edit]

Poster: Shiva Ho Date: Oct 13, 2009 10:16am
Forum: forums Subject: FLASH SUCKS

Your rational is flawed!!! FTP WORKS & Flash doesn't!!!!

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Oct 13, 2009 11:36am
Forum: forums Subject: Re: FLASH SUCKS

then don't use flash! 8-)

there is a "prefer to not use flash?" link
on the uploader (new item) page and
one on the edit item files (for an existing item) page.

--tracey

Reply to this post
Reply [edit]

Poster: Shiva Ho Date: Oct 13, 2009 10:14am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

The "NEW" system has even more flaws & times out repeatedly making it unusable...
Lets FACE it FLASH SUCKS ALWAYS HAS & ALWAYS WILL...
If you can't do it with straight html & java then stop wasting our time & Bandwidth

Reply to this post
Reply [edit]

Poster: Crap From The Past Date: Oct 29, 2009 7:50am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

It seems that you've added an internal check in the checkin procedure that halts the process if you have temporary .pureftpd-upload files still remaining in the directory. That may seem like a good idea, but if the process halts and restarts at some point and creates new .pureftpd files, the old ones are just left there.

I can't delete them with my FTP program ("550 Prohibited file name"), and your checkin procedure now halts ("One or more files (...) appears to still be FTP transferring -- please let it finish first and then reload this page") instead of ignoring them.

Please delete the .pureftpd files from the items "cosmicslop-2004-05-30" and "cosmicslop-2004-04-25", and if possible, please post instructions here on how I can use my FTP program (Filezilla) to delete them.

Thanks!

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Stafftracey pooh Date: Oct 29, 2009 10:13am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

what you say is true as best i can reckon -- our ftpd server is a pain.

suggest workaround:

say you upload to a subdir called "dirt"
that has a pure turd in it now for some reason.

the steps (exact ftp commands vary by client):
rename dirt deleteme
mkdir dirt
mv deleteme/file1 dirt/
mv deleteme/file2 dirt/
...

and then deleteme will vanish automatically some time in the future.
lame/terrible, i know...

--tracey



Reply to this post
Reply [edit]

Poster: Crap From The Past Date: Oct 29, 2009 11:20am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

Actually, that worked pretty well. Hopefully, it will save you the irritation of responding to a lot of similar requests.

To paraphrase what Tracey said and elaborate slightly:

The .pureftp file is stuck where it is. You can't delete, rename or move it.

You can, however, rename the directory it's in.

So take all the useful files you have in the directory (should be all except the .pureftp file), and move them up a level (in Filezilla, drag them to the "..").

Go up a level (click on "..").

Rename the old directory to "deleteme".

Create a new directory with the same name as the old directory.

Change the permissions of the newly-created directory to match those of "deleteme" (Owner: read, write, execute; Group: read, execute; Public: read, execute)

Move all your useful files into the newly-created directory.

The bad .pureftp file is in the "deleteme" directory, which you now leave for dead.

Continue with the checkin procedure.